Ticket: PODAAC-7294

Description

This PR implements support for large file uploads (4GB+) in the MCC application with minimal changes to the codebase. The current implementation works with files up to 1.2GB, but fails with 4 GB+ uploads.

Overview of work done

1. Increased file size limits:

   • Set both API and UI file size limits to 5GB (5,368,709,120 bytes)
   • Updated configuration in checker.wsgi

2. Optimized file handling:

   • Updated file processing functions to use 8MB buffer sizes for better performance
   • Improved memory efficiency by streaming files in chunks
   • Enhanced error handling for large file operations

3. Improved temporary storage:

   • Configured /tmp as the default temporary directory
   • Added directory existence and permission checks
   • Added logging for temporary directory operations

4. Enhanced documentation:

   • Added examples for uploading large files with progress reporting
   • Added documentation for extended timeout options

Overview of verification done

• Tested locally with files of various sizes up to 4.19 GB
• Ensured all checkers (ACDD, CF, GDS2) work correctly with large files
• Checked the ability to generate Web, PDF and JSON reports

Tested in SIT:

• Uploaded a 4.19 GB NetCDF file to the SIT environment
• Verified that all checkers processed the file correctly
• Tested with curl commands using the new progress and timeout options
• Tested the ability to generate Web, PDF and JSON reports

PR checklist:

• Linted
• Unit tests
• Addressed Snyk vulnerabilities
• Updated changelog
• Tested in SIT
• Documentation / User-Guide Updated